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ABSTRACT ' 

The Integrated Competence Seminar (ICS), An Alverno 
College assessment technique used in the outcome-centered liberal 
arts curriculum, is discussed. .The ICS technique allows students to 
demonstrate integration and transfer of learning in three situations: 
oral presentation, group discussion, and the In-Basket exercise 
(decision-making problems). Assessors observe and evaluate 
performance criteria, and provide students with feedback. In the 
1970s, the" In-Basket exercise was the most difficult and the most 
valid in terms of correlation with measures of students' cognitive 
development and other generic ability. Oral presentation showed mixed 
results, and the group discussion correlated with other measures ' 
opposite to the expected directions. ,When. age, background/ and 
college program were controlled, -there were no significant ^ 
relationships ,between the three ICS tasks and other college 
performance measures. Generally, it was found that the In-Basket 
exercise most accurately measured abilities of analysis and 
problem-solving. The results point to the* importance of 
nontraditional assessment techniques, such as In-Basket, and to 
revision of the measure with attention to the links between 
discussion criteria and the social interaction ability it 
(Author/SW) 
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ABSTRACT 

i 

The Integrated Competence Seminar assessment technique allows 
students to demonstrate integration and transfer of learning in 
throe situations: Oral Presentation, Jn-Basket Exercise and Group 
ussion. Assessors observe* and m .te performance against 
■cifisd criteria, and give v 10 students on their perfor- 

mance. A behavioral criteria ci. st permits evaluation of . 

inter-rater reliability, and validation of the technique through 
comparison of quantified assessor jvcigments with % other student 
performance measures and a battery of External % cri terion measures 
administered to students, in a longitudinal study* of college 
outcomes (Mentkowski & Strait , 1983) . * 

Assessor judgments correlated in the 70 f s. We In-". :?ke£ Exercise 
was the most difficult and the most valid in terms of correlation 
with measures of students* cognitive development and other generic, 
abilities. Oral Presentation showed Wxed results, and the Croup 
Discussion correlated with other measures in opposite, to the 
expected directions. When age , background and college program ate 
controlled, there were no significant relationships between the 
three ICS tasks and other college performance measure?, namely, 
number of credits* accumulated and number of competence level units 
achieved. Thus, the In-Basket had some performance validity, the 
Oral Presentation is equivocal, and the Group Discussion had 
relationships opposite to our expections. The finding on the Group 
Disc sion supports our earlier findings with respect to a Social 
Inb ction generic instrument. 

Generally, the effort revealed that the In-Basket exercise most 
accurately measured abilities of Analysis and "Problem Solving. 
The Group Discussion, a measure of Social Interaction, worked less , , 
well. »The study points to the importance of continuing to develop 
hontradi tional assessment techniques like In-Basket, and to 
revise the measure with particular attention to the links between 
Group Discussion criteria and the Social Interaction ability 4 it 
represents. The Integrated Competence Seminar has since undergone 
extensive revisions by a group of faculty specializing in assessment 
design, based in part on the findings of this study. 

j. . e.. 
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VALIDATING ASSESSMENT TECHNIQUES IN AN OUTCOME- CENTERED LIBERAL ARTS 
CURRICULUM: INTEGRATED COMPETENCE SEMINAR 

$ Assessment c^ittee/Of f ice of Research & Evaluation 



The Integrated Compe tency s^lng.^: ' . ' 

^ D iagnostic Asr sessmen_t 

The Integrated Competent Semiriar, an Alverno College assessment t 
technique* provides an opportunity for a student from any discipline to - J 
demonstrate integration of ch e abilities she has developed at^the midpoint 
of her college education. She also shows how she transfers these abilities 
to a new and complex simulated situation she is likely to face as a college- 
educated person. 

Prior to completing the Integrated Competence^ Seminar (ICS)", the student 
has more than" likely been c redentialed on level A of -each competence.. The 
student is not credentialed on her performance on the ICS;cthe assessment 
measures generalization of he r learning to a new situation without the 
pressure to perform for v^^'pion. Thus," the ICS can also b'e viewed as an 
externa* criterion m easur e of a, student's ability to transfer her learning - 
after completing the gene ta l education sequence. 

.This cross-disciplinary sin>Ui..u ion /assesses stuuentfs' integration of 
level 4 of four com P e$enc es : Communications,- Analysis, Problem Solving, and 
Social Interaction. The Integrated Competence Seminar involves the 'student 
in vSrious activities as a member of a decision-making board of citizens who . 
must make a seleftioti ar^tig thre e proposals for distribution of funds for, 
civic improvement. The student takes on the role of a representative of one 
of six. community agencies; .health, culture, recreation, Education, vocational 
training or envirbnmental.^ #tf a^s . - . 

In the ICS , the student engages in *a? series of three exercises over a 



four-hour period. First she makes' an Oral Presentation, After she studies 

a set of background information, she delivers a speech to persuade the 

^ — *i - 

decision-making board to accept her proposal. Next, she completes the- 
Tn-Basket exercise, which derives its name from the in-basket on the manager's 
desk containing ^letters, memos, minutes and reports, all awaiting action. 
She solves problems, sets priorities, analyzes organizes , and makes deci- 
sions* on seven different items- as if she were, a board member encountering 
such situations in her office." In the Group Discussion, the student meets 
with other "board members 11 t:o discuss and ma^e decisions on the final' 
proposal — a common plan for th£ expenditure of funcjs. 

' Of f-eampus' professionals, rather , than Alverno faculty, serve as assessors. 
When off -campus assessors observe students' performance the value of out-of- 
/.^lass learning and assessment experiences are legitimated. Students begin 
to realize that abilities learned in college must at some point be transferred 
to off-caimpus work experiences,' and will be judged by others out of their 
professional perspectives. The assessors thus give an ad<: ■■' 
dimension to student asses .*> 

Assessors observe 'and ev^luate\ each student's performance against 

\ r~ / ' 

specified criteria (oral and group -session^ are videotaped). -Each first 

« . y 

records his or her behavioral observations pertinent to the criteria and then 

meets with, another assessor to achieve consensus and develop a written^ 

j 

evaluative summary. Assessors then schedule individual feedback sessions 
during which an assessor" shares each student's evaluative summary with her, 
discussing strengths and weaknesses in her performance on the ICS. Conse- 
quently, .the ICS i-r^idos diagnost ic info iation for both ^students and 

i • 

faculty. . - ■ i 
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The Integrated Competence Semina r: 
An External Crit erion Measure 

•The major purpose of the Integrated Competence Seminar is, therefore, 

diagnostic; it assists faculty to plan the upcoming professional sequence. 

The measure can also be, usjpd as an 'external criterion measure for curriculum 

validation studies. Prior to using the ICS as an external criterion measure, 

however, quantification of the qualitative behavioral observations' and 

evaluative summary is necessary. Such quantification facilitates statistical 

comparisons of student performance integration across the competences 

assessed at level 4 although the students may have learned their abilities in 

different ways and courses. The ICS total score, can then also be compared 

to other external criterion measures to provide information about the strength 

of the relationships among various student abilities. 

devel oping a Crite ria Checkli^ * 

In order ranslate t assessors 1 qualitative behavioral j observations 

and ev native sumnfery into quantifiable data it was decided to construct a 
listVof) behavioral criteria which could be used as a checklist- for each of 
the three ICS exercises- Oral Presentation, In-Basket and Group Discussion. : 
In Fall 1978, 29 student ICS files were 'reviewed . \ Complete list of 
behaviors recorded by the assessors was generated. There were no behaviors 
recorded by the assessors that were not clearly similar to the instrument's 
behavioral criteria Hoover, 25% of the behavioral, criteria were recorded > 
less than three times across the 29 files. These behaviors were eliminated. 
The remaining 75% comprised the .criteria for the checklists. The question 
as to whether assessors failed to 'identify 25% of the specif led • behaviors 
because the stimuli^ did not, elicit them, the students did not demonstrate' 
them, or assessors were perceptually unaware of thcyn, awaits further 
investigation. # » . .• 



A behavioral criteria checklist was thus designed for each of the three 
exercises: Oral Presentation, In-Basket, and Group Discission." ■ ♦ . 
Assessors respond to each of the criteria by checking the frequency of 
occurrence of student response along the following scale: never, 
at least once, frequently, as "often as possible. This procedure provides a 
score for each criterion, an exercise score for each of the three exercises, 

and ,a combined exercise score describing overall perf ormance on the ICS. The 

' . • ' **» ■ V 

latter score is called "ICS total score 11 (Appendix I). 

Establishing C hecklist Reliabi lity 

Two independent '^valuators then generated quantitative data from the 

29 evaluative summaries using 'the checklist. Intfcr->rater reliability reached 

acceptable levels for ; each exercise: Oral Presentation, r=. 72; In-Basket, 

*i=.75; Group Discussion, r^.?9 (see Table 1). These results supported the 

reliability of tjhe checklist as a quantitative^ measure of qualitative student 

performance dataj. from the ICS and demonstrated that assessors tend bo record 

3 behavioral observations in a- reliable manner, allowing independent ^valuators 
• * * / ' ' m 

to interpret the .squency of. the beliavior's occurrence on a 'quantitative 

checklist.. n • ^ 

Once- the checklist was found to be a -reliable measure, it was instituted 
in the s regular aduinistrat ion of the Integrated Competence Seminar (Fall 
1978). This, meant that the assessors completed the qualitative behavioral 
observations and evaluative summary as well as the quantitative checklist 
frpm direct observation of the student ICS performance. 

Analysis of Student Performance Using ^ : - v 

Quantified Studen t Scores 

— ; = - ' v t 

^ Another sample of students! was then selected for analysis of student 
performance u§ing quantified .student scores obtained from the behavioral 
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TABLE 1 



Inter-rater Reliability of the Integrated Competence 
Seminar Quant. ikaXive Behavioral Criteria Checklist 
Using the Qualitative Evaluative Summary 
as Information Base (n=29) 



\ 



Group Discussion' 



In-Basket f 



Oral Presentation c 



86 
100 

83 

86 

95- 
\ 83 • 
'93 

71 

12 
2 

52 

93\. 

88 

93 

69 

88 
100 •/ 

81 

76 

76 
'98 

71 

•36 

95 

88 
4.00- 

40 

98 

43 \ 



88 
100 

71 

88 
100 

66 
, .95 

80 

52 

3,5 

83 
100 

97 
"83 

64 

7i 

100 
71 
97 
90" 
t 100 
54 
64 
90 
78 

100 
47 

100" 
52 



r =.-79 



Rater 1 Rater. 2 Rater 1 



RateV 2 R^ter 1 R ater 2 



51 . 

95 

70 

47 

74 

79 

86 

42 

40 

70 

4 0 

98 

79 

40 

47 

81 

84. 
, 70 
\100 

67 

98. 

77 

60" 

86 

88 

91 

40 
100 

47 



53 
100 
69 
37 
41 
62 ' 
58 
58 
3J> 
51 
83^ 
•88 
88" 
46 
■51 
76 
72 
39 
97 
74 
10Q 
62 
41 
83 
93 
88 
32 
100 
46 



82 
97 
85 
67- 
91 
67 
"97 
79 
33 
30 
48 
58 
73 
39 
48 
55 
9'4 
70 
73 
88 
97 
82 
42 
.82 
94 
•97 
45 
100 
97 



= .75 



= .72 



Combined ICS ■ . 
Total Scored 

Rate- 1 Pater 2 



90 


22 


23 


96 


29 


29 


84 


24 


22 


93 


20 


. 21; 


90 


26 


23 


54 , 


23 


18 


60 ' 


28 


21 


78 • 


19 


22 


30 


8 


12 


42 


10 


13 


69 


. 14 


2. j 


69 


25 , 


26 


84 ■ 


24 


27 


60 • 


12 


19 


69 


16 


18 


60 


22 


• r 21 


75 


27 / 


25 


51 


22 


16 


90 


. 25 


' 28 


90 


23 


25 


0 


27 .. 


29 


■8 


' 23 


19 


57' , 


14 


16 


37 


26 


23 


93 


27 


26 


78 


29 


27 


66 ■ 


12 ' 


14 


100 


30 


30 


100 


19 


20 



= .78 



a Exercise total score in percent assigned to each student on 
Competence Seminary 



the Integrated 



Raw score 



\ - 
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qriteria checklists. For ty-eight students whq v had completed *Ue Integrated 
Competence Seminar by Fall 1979 and who were part of ^longitudinal sample 

'of« students (entering Alverno in Fall 1977) already participating in a study 

..." 1 " 

of college outcomes, comprised the sample/ 

- Students completed the Integrated Competence Seminar, and assessors 

then observed^student performance, recorded the demonstrated behaviors, came 

to consensus, on the Group Discussion, evaluated each student against the 

behavioral criteria and created an evaluative summary. This time, .the 

assessors also completed the behavioral criteria checklists. 

! The inter-rater reliability of one checklist was first computed. Since 

Group Discussion was the only exercise where two o ( r rri^re. assessors were 

reviewing student performance, we investigated inter-rater reliability tor 

this exercise only. We correlated the checklist rating of each individual 

assessor on Group Discussion with the consensus rating (r/,85). Thus, 

assessors consistently agreed 7 with consensus oh arv overall low performance 

cr overall high performance rating' for each student. 

Each student was first assigned aa exercise "score . The exercise score 

was derived 'by adding the students point s' across all criteria per exercise 

(as cften as possible = 3 points, frequently = 2 paints, ail least once = 

1 point, not at all = 0 points), and then' dividing the student V total by 

the maximum number of points possible for the exercise. " % ^ 

'This yielded a percentage score. For example, the Group Discussion 

exercise consists of 13 criteria. The maximum number of ppints is s<> % 

(13 X 3 = 39). 'ti the studenA.s total number of points for the Grrup 

Discission is 20, then her- percentage score on ,the exercise is 5a%. 

i ' 
Each student was then assigned an, ICS total score. This score, a 

combination of the three exercise scores, was derived by adding the number 
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of points from each of the exercises, and dividing the total by the maximum 
number of points possible across the three exercises. 

The distribution of behavioral criteria percent scores per exercise and 
the ICS total score are presented in Figures 1, 2, 3 and 4.. 

The graphic representation of, the score distributions indicate a bi-m6<Tal 
distribution in the Group Discussion exercise (Figure 1). There were few 
students with low scores and few students with very high scores. However, 
approximately 58% of the students obtained a score between 50-80%. 

In the In-Basket (Figure 2) only 38% of the students obtained a score 
between 50-80% and none obtained a .score higher than 80%. 

T n the Or*i Presentation (Figure 3) 47% of the students obtained a 

\ - 

score between 50-80%. \ 

? The graphs clearly show that the Group Discussion exercise was the 
easiest, and the In-Basket the more difficult one; the Oral Presentation 
fell between the other two. 

Figure 4 shows the total score for all three exercises combined; 
78% obtained a score between- 50-80%, which indicates that some students who 
did poorly on one exercise did wo.il on the others. 

Total student performance scores on each of the exercises were then 
compared via paired t^tests. {'Students performed significantly better on 
the Group Discussion (X = 23.18; SD = 8.68) than they did on the In-Basket 
(X = 17.87; SD = 8.02) (t = 3.3; p <.01). The difference in performance 
on the Group Discussion and the OraJ Presentation was not significant 
(X - 20.79; SD - 6.68) t =■ 1.96; n.s.); the difference in performance on 
the In-Basket and the Oral Presentation was also not significant. ( t « 1.94; n.s.) 
Clearly, students are more likely to perform better in the Croup Discussion 
than they are on the In-Basket or on the Oral Presentation. 
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0-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 

Percent Score 

Figure 1. Relative Frequency IHstribut ion of Students' 
Scores on the Group Discussion Exercise 
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0-10 11-20 21-30 31-40 41-50 ' 51-60 61-70 71-80 81-90 91-100 

Percent Score 



Figure 4. Relative Frequency Distribution of Students* 
Scores op the Integrated Competence Seminar 



Consistency of Students 1 Performance 

Across Exercise Criteria ^ _ , 

To what extent did studenrs perform consistently across exerciser on the 
same behavioral criteria? ^ 

Students 1 performance 6n each ^criterion was compared to examine the pattern 
of differences among the behavioral criteria for each exercise. Criteria are 
grouped under five broader categories: Initiation, Influence, Response, Problem 
Analysis and Decision-Making. Criteria were not identical across each exercise, 
of course-. Group Discussion he 13 criteria, the In-Basket has 12, and the 
Oral PresentatiotKira^ 11. Each student's scale point (0, 1, 2 or 3) per 
criterion wa^entered^lnto a comparison with a similar criterion if at least 
two exercises had a criterion in common. 

Table 2 shows the means and standard deviations for the behavioral criteria 
for each exercise. Several t=tests for paired comparisons were performed, and ar 
entered in Table 3. As shown in Table 3, students demonstrated significantly 
higher performance on Good Impression in the Croup Discussion as compared to 
In-Basket. When In-Basket criteria were compared to Oral .♦Presentation, students 
performed consistently higher in the Oral Presentation on Good Impression, 
Attention and Respect, Confidence, Adaptability, Reading Comprehension and 
Concern for Details. As for Oral Presentation vs. the Group Discussion, 
students performed consistently higher in the Oral Presentation on Confidence, 
Adaptability, Organization arid Decisiveness. 

Table 4' shows the correlation coefficients between the paired comparisons. 

V ■ 

Significant positive correlations were obtained on Good Impression, Attention 
and Respect, and Confidence for the Oral Presentation vs. Croup Discussion 
comparison. Similarly significant positive correlations were obtained on 
Good Impression and Decisiveness in the In-Basket vs. Oral Presentation. 

These results indicate that independent measures of the same behaviors 

• IS 
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tABLE 2 * " ' 

. . ' - \ ' * 

Me-ans (X) ahd Standard Deviations (SD) 

•Per Criterion for Each Exercise 





Group Discussion 


In-^asket 


Oral Presentation 


INITIATION 

Oral Expression- 
Clear Writing 
Initiative 


X SD 




.X SD 


1.71 .80 


1.70 .98 
1.52. , -.92 


i 

2.02 .65 


INFLUENCES 

Persistence 
Good Impression 
Attention #nd Respect 
Confidence v 
Social Effectiveness 


.< 

1.67 .91 
1.86 .86 
1.86 .80 
1.90 .94 
"1.69 .94 . 


1.35 .88 — 

1.56 .96 
1.58 \1.02 


1.85 .71 
2.02 .70 
2.15 .66 


RESPONSE 

Listening 

Flexibility 

Adaptability 

Reading Comprehension 


2.11 .69 
1.61 .96 
1.88 .79 


1.50 '1.03 
1.66 .85 
,1.63 .70 


2.20 .79 
1.93 .76 


PROBLEM ANALYSIS 
Organization 
Concern for Details 
Raising Questions 
Making Predictions 


1.46 .79 
1.62 .81 


J- 

1.41 .76 
1.08 .76 


1.85 .85 
1.93 .74 


DECISION-MAKING 

Considering Alternatives 

Decisiveness 

Value Clarification 


1.60 1.00 

1L62 .90 

\ 


1.22 .72 
1.81 .86 


1.53 1.03 
2.06 .75 
1.62 .84 
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TABLE 3 

Paired Comparisons (t-test) Per 
/ Criterion Across Exercises 



Ax. JU — . 



,.05 

I** p < .01 



20 



• 


Group Discussion 
vs. 
In-Basket 


r 

In-Basket 
vs. Oral 
Presentation 


Oral Presentation 
Group Discussion 


INITIATION 
c Oral Expression 
Clear Writing 
Initiatiye 


t = i - iy 


) 


* — ; 


INFLUENCE 

Persistence * 




t 

t - 3.85** 


t = .177 


Good Impression 
Attention and Respect 
Confidence 

Social Effectiveness 


. _t = 3.2** . 
_t = 1.87 

t = 1 . 


t - -2.87** 


t = -1.85 

. t = -Z.JO 


RESPONSE 

Listening - 

Flexibility 

Adaptabili/y 

Reading Comprehension 


t = 1.44 1 
t = 1 . 3h 


C — J . 04 

V= -2.34* 


t - -2.19* 


PROBLEM ANALYSIS ' : 
Organization 
Pnnpprn for details 
Raising Questions 
Making Predictions 




t = -3.18** 


t = -2.32* 


DECISION-MAKING 

Considering Alternatives 

Decisiveness 

Value Clarification 


t = 1.91 

, t = -1.26 

\ 


t = -1.51 
t = -1.77' 


t = ..23 
t = -2.81* 

■ 
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TABLE 4 



Correlation Coefficients (Comparing Criteria 
Across the Thre'e ^Exercises 



INITIATION 
«Pral Expression 
Clear: "Writing 
Initiative 

INFLUENCE 
Persistence 
Good impression 
Attention and*> Respect 
Confidence 
Social Effectiveness 

RESPONSE 
Listening 
Flexibility — ' 
' Adaptability 
Reading Comprehension 

PROBLEM ANALYSIS 

- Organization 

Concern for details 
Raising Questions 
Making Predictions 

DECISION-MAKING 

Considering Alternatives 

Decisiveness 

Value Clarification 



G^up Discussion 

, * -vs. ( ' 
in-Basket 



,12 



.26 
.12 



.089 
.009 



-.067 
.20 



* p < .05 
** p < .01 



In-Basket' 
vs. Oral 
Presentation 
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.27 
.28 



.22 
.17 



-.067 



.13 
.27* 



Oral Presentation 

"vs. 

Group Discussion 4 



.43** 
.47,** 



.12 



.096 



.19 
.21 




- ' ' \ ■' ' . ' ' ' ' 

were not highly consistent across exercises, which suggests that certain skills- 

.may be a function of th,e specific situation in which they are demonstrated. 

* . i ' ' 

J3ne also wonders if the assessors do interpret the same capabilities differently 

/ 

in different situations, or even if the stimuli are equivalent ijt eliciting ,! 
similar behaviors. That is, are the fc t^haviors truly comparable? This questiort 
has implications for our understanding of students' ability to transfer comp^-, 
tences *to different situations. 



I 

In addition,'. we -intercorrelated all criterion score,s (n=36) » disregarding 

^ 1 
exercise. The correlation matrix indicated clusters of higher positive 

correlations between In-Basket criteria and Oral Presentations/criteria. For * 

4c. 

example^ all three criteria,- Good Impression, Attenfriop and Respect, and' 
Confidence in the In-Basket correlated highly with ^ach other, and formed a 
cluster with similar abilities in Oral Presentation. All criteria associated 
with the broader category of Decision-Making which include Cor -idering Alterna- 
tives, Decisiveness and Value Clarification were highly intercorselated in the 
In-Basket and formed a cluster with similar criteria in the Oral Presentation,. 
,^These criteria under Decision-Making are also significantly correlated with 
Flexibility and ' Adaptability within the In-Basket exercise. These correlations 

suggest that situations where a person acts alone may elicit th^ same abilities. 

& . ...» 

The In-Rasket and Oral Presentation are individual exercises whereas the Group 
Discussion adds an interpersonal dimension to the same skill. Table 4,- - 
however , shows positive correlations between Oral Presentation and Group . 
Qiscussion on behaviors which reflect personal style. Thus, personal style may 
'be consistent irrespective of the exercise mode. v 

Still another issue of interest in the analysis of student performance on 
the Integrated Competence Seminar is the extent' to which certain behavioral fl 
criteria represent behaviors t fiat" are easy vs. difficult for students to perform 
after two years in college. Which behaviors are performed by most students? 



^his 36 X 36 intercorrelat ion matrix is available from the Office of 
Research and Evaluation. 

\ 
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Thus/ the next step w^s to further examine the distribution of student 
performance on each criterion. As stated earlier, criteria are grouped under 
five broader categories: Initiation, „ Influence, Response, Problem Analysis and 
Decision-Making. Group Discussion has 13 criteria, the In-Basket has 12, and 
the Oral Presentation has 11. .Criteria were not identical across each exercise, 
of course. Table 5 shows the percentage of students (n=48) who performed at 
each poipt on the scale per behavioral criterion for each exercise. ^ 

An examination of Table 5 shows thab the students are rather evenly 
dist\ibuted across three of the points on ttje scale for most criteria, 
namely, As Often as Possible, Frequently, and Ats^aat Once. In general 
small percentages of" students responded at* the Noi; at All point on the scale. 
These distributions conf irm that the Integrated Competence Seminar may -serve 
as an external .criterion measure that will provide variability in performance 
that can be compared to that of other students, and to performance on other 
measures. Since relatively few percentages of students respond in the 
"H$¥t at All category, we may conclude tliat' students do demonstrate the 
beha\^prs as opposed to not demonstrating them.^That is, they have achieved . 
the behaviors to some measureable degree. p> * 

There are, however, some behaviors that seem less likely to be demon- 
strated, and seem to stand out from al^the rest. These are important because 
we may wish to review the stimulus to examine the extent to which the measure 
actually elicits or provides opportunity for demonstrating these .behaviors. 
We might also examine the extent to which we would expect that these j 
behaviors ai*e likely, to be exhibited after just two years in college. These 
criteria also may indicate some -excellent discriminative behaviors that 
would add to the value of the technique for comparison purposes as an 
external criterion measure. We chose, somewhat arbitrarily, the figure of 
10% in the "not at all" scale point at a cutoff for identifying criteria 

23 
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TABLE 5 

Percentage of Students (n=48)^ Who Performed at Each 
Scale Point Per Behavioral, Criterion for Each 
Integrated Competence Setriinar Exercise 



INITIATION ° V V^ 


GROUP DISCUSSION 


IN-BA^KE?' 


ORAL PRESENTATION 

. i... 


As often 
as pbssible 

Frequently 

At least 
once 

Not at all 


As often' 
as possible 

Frequently 

At leaa* — -* 
once -C 

Not at all 


As ofien 
as possible 

Frequently 
< 

"At least 
once 

Not at all 


Oral Expression 
Clear Writing 
Initiative / 


-- -- ' -- 

21. 6 a 3.7.8 37.8 2.7 


22.9 39.6 22.9- 14.6 
12.5 43.8 27.1 16.7 


22.2 57.8, 20. (f — 


, s 

INFLUENCE ' ' ^ : '» . 


Persistence , 

Good Impression . % 

Attention and Respect 

Confidence 

Social Effectiveness * 


24. 3 43.2 24.3 8.1 
1.0.8 9 18.9 5.4 
13 . T 5 59^27.0 — 
18.9 51.4 2V>sl^5.4 
27.8 33.3 27.8 11.1 


6.3 43.8 29.2 20.8' 
18.8 33.3 33.3 14.*6 
20.-9 34.9 25.6 18.6 


18.8 47.9 33.3 ~. 
25.5 51.1 23.4 — 
30.4 54.3 15.2. — 



RESPONSE 



Listen ill}; 
Flexibility 
Adaptability ' 
Reading Comprehension 


21.6 35.1 40.5 2.7 

29.7 45.9 21.6 2.7' 


20.8 27.1 33.3 18.8 
12.5 54.2 20.8 12.5- 
6.4 57.429.8' 6.4 


39.645.8 10.4 4.2 
23.4 48.9 25.5 2.1 


PROBLEM ANALYSIS . - . . " 


Organization 
Concern for Details 
Raising Questions 
Making -Predictions 


8.1 48.6 37.8 5.4 

13.5 37.8 37.8 10.8 

/ 


4.2 45.8 37.5 12.5 
4.2 20.8 54.2 20.8 


22.9 45.8 25."0 6.3 
19.6 58.7 17.4 4.3 


*- ..... 

DECISION-MAKING / . - 


Considering Alternatives 

Decisiveness 

Value Clarification 


18.9 '45.9 24.3 10.8 
16.'2 40.5 37.8 5.4 


7 

2.1 35.3 50.0 14.6 
22.9 41.7 29.2 6.3 


19.1 36.2 23.4 21.3 
2$. 2 50.0 18.8 2.1 
16.7 35.4 .41./ 6.3 



Scores in percent. 
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that niay be such discriminative" criteria. That is, if 10% or more of the 
students responded at the Not At All scale point, ve felt the behavior would 
be one that students found to be more difficult to demonstrate then others. 

An examination of Table 5 indicates that there are three' criteria in the 
Group Discussion that may be difficult: Social Effectiveness, .Raising 
Questions and Considering Alteratives. There are ten such criteria in .the * 
Ih^Basket: -Clear Writing, Initiative, Good Impression,. Attention and Respect, 
Confidence', flexibility, Adaptability, Concern for Details, Making Predictions, 
and Considering Alternatives . For the Oral Presentation, there is one such 
criterion:' Considering Alternatives. 

These results indicate' that 10 of the 12 criteria on the In-Ba/sket show 
at least .10% of the students responding at the Not At All scale point: the 
„range is fro© 12.5% to 20.8%. This may indicate .that the In-Basket exercise 

» * 

score does discriminate high and low performing students rather well. 

~fhe finding that not ail students performed all of the criteria in the 
In-Basket is confirmed when we examine the number of items completed in *the • 
In-Basket exercise. The In-Basket contains seven items in all; the students 
are' required to organize their time, set priorities, and kt tempt to complete 
all items. Table 6 shows the percentage .of students completing a certain 
number of items. 

■** » 

> ■ * TABLE 6 

V 

Percentage of Students (n=48) Completing Each 
v of the Seven Items in the In-Basket Exercise 

Number of Items Completed { Percentage 

None of the seven 0.00 

One of seven 87.5 

Two of seven 95.8 

Three of sevgn • 83.3 * ' 

F ur of seven 72.9 „ , 

Five of seven 62.5 

Six of seven 50.0 , , 

Seven of seven 50.0 



Since 50 percent of the students completed all, seven. itemed and \ 
50 percent did not, the numbs.- -r of 'items completed in the In-Basket, e'xercise 
can be considered an additional discriminative measure. The fact that t 
number of items completed shows .variation, as do the behavioral criteria 
for the In-Basket, suggests that the number of items completed also may be 
considered 'as a measure of performance on the In-Basket. 

If the number of' items completed would be used, the instrument designers x 

mignt examine the underlying meaning of this^measure in relation to the N 

behaviors the In-Basket is designed to elicit- For example, how does the. 

• "* ■ t 

number of items completed relate to criteria such>as organization, setting 

*\ ^ 

priorities and difficulty of items? 

In order to further examine the extent to which each 6'f the behavioral 

criteria discriminate between higft and low performing atu^ents, we correlated* 

a ' ■■ * 

each student's score per criterion with each student's corresponding total 

' . . • •..„•(•" 

exercise score (see Table 7). If all criteria correlate significantly 

, <P * ' . 

positive with the corresponding exercise total scor.e the criteria are assumed 

to be nondiscriminat ive. 

An examination of the correlation coefficients in Table\7, shows^that all 
criteria within each exercise correlate significantly positive with the corres- 
ponding exercise, .score. Thus, we may not ^conclude that the criteria mentioned 
earlier (at least 10% of the students did not demonstrate them) are f good J 
discriminative items. „ # , 

An alternate interpretation of the .correlation matrix suggests that 
assessors,, while rating student performance," may not, discriminate among the 
criteria. That is, assessors may form a judgment : about a person's overall 
performance using _ne behavioral observations he or she records to create 
this holistic expert judgment. When the quantitative checklist: is completed, 



TABLE 7 



Correlations 


Between Each 


Exercise "Behavioral 






Cr iter ion 


and Total Exercise Score (n=Al) 








Group 




Oral 






Discussion 


In-Basket 


Presentation 


INITIATION 










Oral Expression 








.81 


Clear Writing 




7 L 




— 


Initiative 


.72 


. 79 




— 


INFLUENCE 










Persistence 










Good Impression 




. 86 




.84 


Attention and Respect 




. 82 




.80 


Confidence 


,85 


. 79 




.75 


Social Effectiveness 


.86 






..... . . 


RESPONSE 










Listening 


7 S 






-- 


Flexibility 


. 76 


.90 




— 


Adaptability 


.85 


.86 




.72 


Reading Comprehension 




.84 




.81 


PROBLEM ANALYSIS 










Organization 


- 79 






.85 


Concern for. Details 




.78 




.76 


gaising Questions 


. 66 








Making Predictions" 




. 70 






DECISION-MAKING 










Considering Alternatives 


.73 


.65 

A 




.76 • 


Decisiveness 


,69 


.76 




.83 


Value Clarification 








.78 


NOTE: All correlation 


coefficients 


are significant at the 


.001 


level . 
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however, this expert judgment colors judgment on each of the individual 

criteria. This is known as the "halo effect." That is, if an assessor is 

likely to judge a student's overall performance as high, the assessor is 

also likely to judge the student's performance as "high" on each of the 

separate checklist criteria. This interpretation is not only supported hy 

the significantly positive correlation coefficients between each exercise 

criterion and the corresponding exercise total score. We also obtained an 

intercorrelation matrix for each exercise that shows the correlation of each 

criterion with every other criterion. These intercorrelation matrices are 

depicted in Tables 8, 9, and 10. Examination of these matrices shows 

positive significant correlation coefficients for almost each comparison of a 

criterion with each of the others in the exercise. This pattern also seems 

to indicate that assessors are less likely to differentiate between students 

\ 

on a single criterion; rather, they do differentiate students on an overall 
exercise score, or on the ICS total score. One might speculate that 
assessors use the specific criteria to call forth the behavioral observations 
they made earlier, but that when they make a judgment per criterion, the 
judgment is affected by the overall concept of a student's performance 
(halo effect). Thus, students who receive an overall high rating are als6 
rated high on all criteria. 

Although we have some evidence for the operation of a halo effect in 
the criterion rating given by assessors, assessors do consistently agree 
with each other on an overall low performance or an overall high performance 
rating for each student. The correlation between the individual assessor 
rating and consensus for the Group Discussion exercise was .85 (see page, 6). 
Further, assessors do distribute, their ratings across the four scale points. 

In sum, analysis of the specific criteria suggests that assessors make 
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TABLE 8 

Intercorrelations Within Group Discussion 





IGR0UP1 


group: 


GR0UP3 


Initiative 


groupi i.oooo 
( d 

S« .001 


.6201 
( 42) 
S« ,001 ' 


.4618 
( 42) 
8" ,001 


Persistence 


OSBTL . *U0l 
1 42) 
S- ,001 


i.oooo 

( 43) 
S" .001 


.6032 
1 43) 

Si ,001 


Good Impression 


GR0UP3 ,4619 
( 42) 
,001 


.6032 
{ 43) 
8" .001 


1,0000 
I 43) 
9* .001 


Attention 
uml Respect 


mm ,4947 

( 42) 
6» ,001 


,6139 
I 43) 
S" .001 


.8992 
( 43) 
B- .001 


Confidence 


GROUPS ,5964 
( 42) 
S* .001 


.6489 
( 4J) 
5" ,001 


,0890 
( 43> 
S" ,001 


Social 

Kffectiveneii 


GR0UP6 ,6269 
( 41) 
• 8' .001 


.6906 
( 42) 
6« ,001 


,5670 
( 4?) 
6* ,001 


Listening 


GK0UP7 ,4900 

{ 4:) 

5- .001 


.5418 
( 43) 
S» .001 


.6619 
( 4J) 
8* .001 


Flexibility 
t 


0W8 ,4332 

( 42) 
. .S" .002 


,3846 
( 43) 
8- .001 


,3206 
1 43) 
S- ,001 


Mutability 


GROUPS ,4434 
■ 8* ,001 


.6331 
( 43) 
S> .001 


.3333 
I 43) 
8- ,001 


Organization 


6R0UPI0 ,4492 
I 42) 
8* ,001 


,5349 
( 43) 
S» .001 


,5821 
( 43) 
5* .001 


Nilfllng Questions 


CROliPil ,4344 
1 42) 
S« ,001 


.4374 

( 43) 
S« ,002 


.J642 
( 4U 
8« .008 


Considering 
Alternatives 


0ROUP12 ,4220 
< 42) 
S' .003 


,4770 
( 43) 
S> .001 


.4030 
( 431 
9" ,004 


Decisiveness 


GkOUPlJ ,44M 
( 42) 
5< .001 


,4 r ( 4A 
( 43) 
.001 


,4514 
( 43) 

s» »ooi 
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GROUP 4 


mm 




OMUP7 


GROUPS 


,4947 
I 42) 
S> ,001 


,5964 
( • 42) 
S' ,001 


,6269 
( - 41) 
S» ,001 


,4908 
( 42) 
9" .001 


.4332 
( 42) 
S" .002 


.41)9 
( 43) 
S- ,001 


.4489 
( 43) 
S> .001 


,6906. 
( 421 
S- ,001 


.5438 
( 43) 
3« ,001 


.,2644 
( 411 
9« ,001 


,8992, 
( 43) 
gi .001 


,8890 
( 43) 
Si .001 . 


.3670 
< 42) 
*■ ,001 


.6619 
( 43) 
8* .001 


,5206 
I 431 
. 8« .001 


1.0000 
( *3l 
5- ,001 


,0896 
( 43) 
5- ,001 


.6094 
( 42) 
S' ,001 


,5816 
( 41) 
S* .001 


,3878 
( 43) 
S< ,001 


,0896 
( 43) 
S> ,001 


1,0000 
( 41) 
S» ,001 


.63^2 
I 42) 
Si ,001 


.5940 

( 4.1) • 
5- .001 


,5209 
( 43) 
S- ,001 


,6094 
I 42) 
8' .001 


,63*2 
( 42) 
G< .001 


1,0000 
( 42) 
8' ,001 


.5618 
{ 42) 
G> ,001 


,6942 
( 42) 
6' .001 


,MI6 
( 43) 
S" .001 


,5940 
( «3) 
S" ,001 


,56' A 
( 42) 
5« .001 


i.oooo 

( 43) 
Si ,001 


* ,5947 
( 43) 
S> ,001 


.5878 
< 43) 
5' .001 


,5209 

1 43' 
S* .001 9 


.6942 

( 4?l 

5- ,0M 


,5947 
( 43) 

5* ,001 


1,0000 
( 41) 
Si ,001 


.4966 
( 43) 
S> .001 


,6196 
( 43) 
6i .001 


.7720 
( 42) 
S* ,001 


,5417 
( 43) 
8* .001 


,6009 
I 43) 
S- ,001 


.5121 
< 43) 
5« .001 


,4244 
( 4Ji 
3> ,001 


,4855 
( 42) 
5' ,001 


,6285 
( 4]) 
Si ,001 


.4373 
( 43) 
,5" .002 


.4263 
< 43) 
S* .002 


.3851 
( 43) 
5* .0)1 


♦4844 
( 4?) 
Si ,001 


.5374 
< 4]) 
S« .001 


.5433 

< 43) 

s« ,om 


, 343*1 
( 43) 
B« .012 


,4116 
( 43) 
8* ,003 


'.6363 
( 42) 
6i ,001 


.3736 
( 41) 
Si ,007 


.4007 
I 43) 
S« .601 


,4527 
( 4JJ 
S* .001 


.6010 
( 4J) 
S' ,001 


,4921 
( 4,) 
S- .001 


,3740 
( 41) 
Si .007 


,3314 
( 43) 
S< .015 



GRGUP9 


GRDUP10 


GROUP 11 


GR0UP12 


6MUPU 


.6634 
. 1 42) 
Si .001 


1-4692 
( 42) 
8° .001 


,4564 

4- 42) 
Si ,001 


.4228 
< 42) 
8» ,003 


,4474 
4?) 
Si .001 


• ,6331 
( 43) 
S» .001 


■ .3368 
( 43) 
S- .001 


,4374 
( 43) 
8i .002 


.4770 
( 43) 
8* ,001 


.4546 
{ 43) 
S- .001 


,5333 
( 43) 
8« ,001 


1 

,5021 
( 43) 
. Si .001 


fin 

.3642 
i 43) 
U ,408 


• 40!0 
i 43) 
8< .004 


♦ 4frJ 1 

< 4]) 

• &•■ cOOI 


.4966 
( 43) 
Bi .001 


.3)21 

( 4.11 
S* .001 


4 43) 
»• .002 


Til, 

.( ,. 4,11 
8* ,012 


,4.)27 
( 4j) 
Si ,r>0l 


,6196 
( 43) 
Si .001 


,6266 
( 43) 
Si .001 


,3851 
( 43) 
Si .005 


,4116 
( 43) 
Bi .003 


40I* 
( 43) 
Si .001 


.7720 
( 42) 
8' .001 


,4853 
( 42) 
Si ,001 


,6064 
( 42) 
•8' .001 


,6363 
( 4?) 
^ I* ,001 


< 42) 
E- ,001 


.5417 
( 43) 
5* .001 


,4203 
i 43) 
5' ,001 


.3374 
< 43) 
5* .001 


,3736 
< 43) 

Bi .oor 


i «J 
5' ,00? 


,6009 

( 43) ' 

Si ;ooi 


,437] 

( 43) 
S* .002 


,3433 
( 43) 
8« .001 


♦4107 
1 43) 
8* .001 


.3314 
i 43) 
S» ,015 


1.0000 
< 43) 
Si .001 


.4324 
( 43) 
S* .001 


.5562 
< 43) 
-Si #001 


4W 

( 41) 

8' ,001 


,4'U 
< 431 
S« M 


.6324 
I 43) 
Si .001 


1.0000 
( 431 
Si .001 


iff Al 

,4W 

.( 43) 

Si .001 


,6818 
( 43) 
Si .001 


( 43) 
Si ,001 


.3542 
1 41> 
S* ,noi 


.4547 
( 43) 
S« .001 


- 1,0000 
( 4!) 
Si ,001 


,4042 
( 43) 
«• .00| 


.2604 

$■ .044 


.6590 
< 4J> 
8* .001 


.6810 
( 43) 
* Si .001 


,4042 
( 43) 
.in ,001 


1.0000 
9i .001 


450 i 
« «J 


.4*15 
< 41) 

,00t 


.6780 
( 43) 
5» .001 


,2604 
( 41) 
Si ,046 


4104 
1 .43) 
8« ,001 


I 43) 
Si ,ooi 
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TABLE 9 

Intercorrelations Within In-Basket 

8ASKE15 BASKET6 l ( A5KCT7 MSKfTII 



to 



MSKIT9 W5I.ET10 BASM'Tll 



f*1 »»» 



Writing 
Initiative 



Cood 

hpreasion 



Attention 



Confidence 



flexibility 



Reading 



Concern for 
Details 



Making 
Predictions 



Considering 
Alternatives 



Decisiveness 



JI ERLC 



4 AAAft 
1 . (WmHJ 


C ,A70 


,4B44 


,5323 


,7129 
( 43)' 
5= .001 


,5009 

> *n\ 


■ .6600 
( 4B) 
s-- ,001 


,6160 

I 4/1 


,4162 
( 40) 


,3133 
( 48) 


( 4n> ' 

5- .001 


( 4fl) 
5= .001 


( 48) 
S« .001 


( 4fl> 
5= ,001 


\ MO/ 

.001 


S= ,001 


S» .002 


S' .015 


.jo/0 
( 48) 
5* .001 


( 48) 
5= .001 


,6020 
( 48) 
5* ,001 


,5003 
( 48) 
S* ,001 


,6307 
( 43) 
S= ,001 


,6374 
( 48) 
Ss .001 


,5730 
( 40) 
Ss ,001 


,6199 

\ Hf ) 

S« .001 


i 5jBB 
S* ,001 


,50B7 
( 481 
S« ,001 


( ft* 

s- .001 


( 4(1) 
S* ,001 


1.0000 
( 4W , 

s= .oqi\, 


,6320 
( 48) .. 
S= ,001 


,5577 
( 43) ■ 
S» .001 


,8023 
( 48) 
S* .001 


,6331 
( 40) 
Ss ,001 


,6847 
( 47) 
S» ,001 


" ,6539 

/ AH) 

S* .001 


,6435 
( 48) 
S* .001 


5 ,'»0! 


,5003 
( 4R) 
5= ,001 


.6320 

i 48) 
S= ,001 


ubooo 

( 4P) 
S= ,001 


,7307 
( 43) 
S= ,001 


,6303 
( « 48) 
S= .001 


.5645 

( 48) 
S= ,001 


,6493 

( 1/ J 

S= ,001 


.5673 

( 4R) 
S= .001 


,4811 

( 48) 
S= ,001 


7P9 
( 41) 
S* .001 


.6307 
( 43) 
S* .001 


.5577 
( 43) 
S= .001 


,7307 
( 43) 
S« ,001 


1,0000 
( 43) 
S« ,001 . 


.5428 
( 43) 
S« .001 


.6578 

/ Al\ 

I 4j J 

S* .001 
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TABLE 10 

Intercorrelations Within Oral Presentation V ^ J 
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reliable judgments about the overall exercise score on the Integrated Compe- 
tence Seminar, and recognize, evaluate and record the individual differences 
between students on this overall rating. It may be that assessors use the 
specific criteria as guides to 'what to look for," to record behavioral 
observations, and to detail the various aspects of the student's performance 
that comprise the general overall concept the assessor forms about the 
student's overall performance on the Integrated Competence Seminar. Just 
as the student is expected to integrate her performance on this simulated 
assessment technique, so the assessor integrates the information from the 
pattern of behavioral observations he or she makes, to form a highly 
reliable judgment that discriminates students' individual differences in 
performance. 

The forty-eight students assessed also demonstrated that, as a group, 
almost all performed adequately on the instrument. 

■ -\ 

Validating the Int egra ted Competence 
Seminar in Relation to Other Student 
Performance Measures in the Lon gitudi nal 
S tudy 

As mentioned earlier, the forty-eight students comprising the sample 
used for the preceding analyses were part of a longitudinal sample of students 
participating in a study of College outcomes. The Integrated Competence 

r 

Seminar was employed as a generic measure of student performance in Alverno's 
efforts to validate its outcome-centered curriculum (Mentkowski & Doherty, 
1977, 1983; Mentkowski, 1980). Several strategies for validating outcomes 
were developed into a comprehensive validation ^odel . The model incorpo- 
rates various research and evaluation methods, with the ultimate goals 
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of establishing program validity, contributing .to program development, 
and developing a picture of adult learning and development that can be used 
to improve instruction and assessment in liberal education settings. 
..Figure 5 displays the several components of the validation model. The 
results reported be Tow link performance on the Integrated Competence 
Seminar assessment to other internal college performance variables (e.g., 
competence level units and credit hours) , to performance characteristics, 
and to the external criterion measures of human potential. Other papers 
present results linking the human potential measures to the performance 
characteristics rating (Assessment Committee/Office of Research and Evalua- 
tion, 1983) and to the internal college performance measures (Mentkowski 
& Strait,* 1983). 

Longitudinal data Were collected on two consecutive c Lasses entering 
in 1976 and 1977. While the ICS was administered on a single occasion, 
at the midpoint of the student's education, the measures of human poten- - 
tial, identified in Table 11, were administered on three occasions: at 
entrance, two years later, and three and one-half years after entrance. 
Thk second longitudinal assessment roughly coincided with the midpoint F 
administration of the ICS so that the relationships between ICS performance 
scores and measures of human potential can be examined in the three 
temporal relationships illustrated in Figure 6. Figure 6 also indicates 
when other college performance variables were collected. Competence 
level units, credits and number of semesters were tabulated at the time 
of the second and third longitudinal assessments; the Six, Performance 
Characteristics rating for one student cohort occurred twice, at the end 
of t;he third and fourth academic years of the study; for the other cohott 
three times, at the end of the second, third and fourth years. 
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Table 11, 



Measures of Human Potential From the Longitudinal Study 



HUMAN POTENTIAL MEASURES: COGNITIVE DEVELOPMENT 



Test of Cognitive Development (Renner et al., 1976; after Piaget) 

By having a studewt work a series of problems and provide reasons for his 
or her answers, this instrument measures a student's cognitive activity 
based on Piaget ? s stages of cognitive development. 

Sentence Completion Test (Loevinger, et al*, 1970) 

This instrument provides a measure of an individual Is stage of ego 
development. "Ego" here is defined as one's "style of life," the unity of 
personality, individuality, the method of facing problems, opinion about one's 
self and the problems of life, and the whole attitude toward making choices 
in all life spheres. 

Defining L Issues Test (Rest 1979) 

Rest's instrument (based on Kohlberg's theory 1 of moral development) 
provides a measure of an individual's moral development in a recognition task 
by analyzing the relative importance attributed by a person to principled 
moral considerations. A person attributes importance to several reasons 
given for resolviag a particular moral dilemma, and then rank orders them. 

Measur e of Vocational , Education al , and Personal Issues (KnefeTkamp, 1974; 
Widick, 19/5; now titled: Measure of Intellectual Development; after Perry) 

This measure of the Perry scheme of intellectual and ethical develppment 
asks students to write three essays on their best class, a major decision and 
v their career. It assesses the progress the college student makes toward * 
movement on the Perry scheme. * 



HUMAN POTENTIAL MEASURES: GENERIC ABILITIES 

Test of Themati c Ana lysis* (Winder, 1976) 

This instrument consists of two sets of stories students are asked to 
compare themat ically . „ This "thematic analysis" is scored according tt^aJ^ 
twelve categories of critical think^og. This test is based on an understanding 
of cognitive development defined as the ability to analyze new information and 
to synthesize new concepts based on this "information , and reflects the ability 

'.to integrate information into one f s own cognitive structure. As the cognitive 
structure grows, so does .the ability to think critically, to make a cogent 

-argument and to reason inductively. f 



^Available from MfcBer and Company. 
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.Table 11. continued 



Picture Story Exercise* (Scored for Stages of Adaptation (Stewart , 197/), 
SelT-oVfrnition (Stewart & Winter, 1974); and Achievement (McClelland, et al., 
1953, Affiliation (Atkinson, 1958), and Power (Winter, 1973) motives.) 

This instrument, modeled on the Thematic Apperception Test, may be 
used to assess a variety of abilities. The instrument requires the student 
to write narratives to six pictures. One is "self -def inition" which encom- 
passes the way one thinks about the world and one's self, the way one reacts 
to new information, and the way one behaves (Stewart and Winter, 1974). 
People with high cognitive initiative are not only able to think iclearly , 
but also to reason from problem to solution, and to propose and take effective 
action on th^eir own. This instrument is also used to assess Need for Achieve- 
ment (McClelland, et al. , 1953), Affiliation (Atkinson, 1958), Power (Winter, 
1973), and Stages of Adaptation^, a measure of ego development (created by 
Stewart, 1977). V 

Wat s tan-Gl aser Cr i t i cal_ Jh Aj]^I^..APPJA^ tf i i -- 1 - (Watson and Glaser, 1964) 

" This instrument measures several components of critical thinking: 
Inference, Recognition of Assumptions, Deduction. 

Le a r ning_ Style Inve ntory * (Kolb, 1976) 

The Learning Style Inventory is a measure of individual learning styles 
which affect decision-making and problem-solving. The four styles are 
Concrete Experience, Reflective Observation, Abstract Conceptualization, and 
Active Experimentation. The instrument requires the student to rank order 
descriptive statements about her mode of learning. 



^Available from McBer and Company. 

^For a more detailed description of the measures and their 
use in the longitudinal study, see Mentkowski, M. & Strait, M. 
A! Longitudinal Study of Student Change in Cognitive Development- 
and Generic Abilities in an Outcome-Centered Liberal Arts Curriculum . 
Milwaukee, Wi : Alverno Productions, 1983. 
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Note. See Figure 5 for overview of components of the program validation model with measures, 
Student Perspectives Interview (S*> T ) data were collected on a subsample of students 
participating in the administration of the Human Potential Measures (HPM), but all complete! 
the Attitude Survey (AS) and Careering Questionnaire (CQ). All Weekday College students completed 
the Integrated Competence Seminar (ICS) and were rated by faculty on the Six Performance 

Characteristics (SPC). «- 

Figure (>. Design for the administration of human potential measures and student perception measures for 
longitudinal and cross-sectional studies of student outcomes. 
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The major difference between the data analyzed and presented in this 
report concerning the. human potential measure outcomes, and the data presented 
in the detailed study of student change (Mentkowski & Strait, 1983), is 
the absence, in the present study, of the "Weekend College Cohort . The 
alternative Weekend College timeframe attracts older, working women whereas 
the Weekday College timeframe attracts the traditional age, less experienced 
college student. Consequently , the student population represented in this 
report is much more homogeneous with respect to age and other background 
and program characteristics. 

The two consecutive entering classes comprising the present sample^cf^y 
be described in similar terms. At entrance, approximately 90% of the sample 
were single, about 80% were 17 to 19 years old, and about 70% identified 
themselves as Catholic. In terms of measured program variables, about two- 
thirds were commuting, and virr.uaily all were attending college fulltime. 
Two-thirds of the students identified themselves at entrance as majoring 
in nursing, with the remainder scattered through Fine Arts, Education, and 
traditional liberal arts majors. 

Analysis o( ICS Scores for the 
Weekday College Longitudinal , Sample 

The combined classes provided a longitudinal sample of 136 students. 

Integrated Competence Seminar scores were available for 131 students. The 

distributions of students 1 scores per exercise and the ICS total score are 

presented in Figures 7, 8, 9, and 10. As on the previously described case 

(Figures I, 2, J, and 4), those graphs show that the highest scores were 

obtained on the Croup Discussion exercise, am! the lowest scorer, were obtained 

on the in-Basket exorcise. 
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Q-10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 

Percent Score 

Figure 9 . • Relative Frequency Distribution of Students' Scores on the 
Oral Presentation Exercise For the Longitudinal Study 
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Percent Score 

Figure 10. Relative Frequency Distribution of Students 1 Scores on the 
Integrated Competence Seminar for the Longitudinal Study 
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In the larger sample, there were more high scores on each exercise, 
but the total score distribution discloses that there were few students 
who received the highest scores on all three exercises. Correlation coef- 
ficients between the various exercises were all statistically significant. 
For In-Basket and Oral Presentation r = .399, £ < .001; for In-Basket and 
G^cup Discussion r = .260, £ < .001; for Oral Presentation and Group Disc^us- 
Vion r = .444, p < .001. This would support that the exercises elicit 
different skills, assuming reliable measures, but that they share similar \^ 
aspects as well. 

Relations hips B etween ICS and 

S t udent_ and ^ ogram^iaract erij^te s_ 

There were few statistically significant relationships between ICS 

X 

performance scores and measured background and program variables. Speci- 
fically, there s were no significant correlations between ICS performance 
and age, parents' education or occupation, or high school grades. There 
were small but significant correlations between some ICS exercise scores 
or total score and variables recording religious affiliation and marital 
status. Students who were not identified as Catholic or who were not 
single tended to receive higher performance scores. Scores on the In-Basket 
exercise and the Total Score were significantly correlated with religious 
affiliation (r = .203, n - 129, p == .011, and r - .179, n = 129, p_ =^D21, 
respectively). Marital status was correlated with performance scores on 
the Oral Presentation exercise (r = .160, n = 131, P. - -034) and with Total. 
Score (r = .172, n = 131, p_ - .025). 

Considering the highly dispropor t ional representations of subgroups and 
the hi^intercorrelations among background variables, it would be inappro- 
priate to claim any theoretical association between religious affiliation 
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or marital status and performance on the ICS. The correlations suggest 
rather a demographic factor .based on the local context of Alverno College • 
In the Weekday College, the typical student was of traditional age, single, 
Catholic, and a fulltime student. The observed correlations between back- 
ground variables and performance on the ICS indicate that a student who 
was atypical with respect to the local context was likely to receive 
higher scores. 

Among program variables, fulltime status was correlated with Oral 
Presentation and Total Score, but this relationship was redundant with 
relationships observed between ICS scores and background variables. There 
was a small but significant relationship between Oral Presentation scores 
and selected major field (R = .314, F (6,123) = 2.24, £ - .044). Students 
majoring in behaviora^ sciences had the highest mean score, and students 
majoring in art the lowest. The disproportionately large group of nursing 
• majors averaged below behavioral science, natural^cience , and education 
majors, but above art, arts and humanities, and undecided majors. There 
were no significant correlations between ICS performance scores and studen^.? 
residence. 

The two entering* class cohorts were not significantly different from one 
another on ICS performance scores; in fact, the averages were nearly identical. 
As partial confirmation of the typical/atypical dichotomy suggested above, 
students from the pooled entrance cohorts were reclassified as typical r 
if they were single, Catholic, and fulltime; atypical if they were not all 
three. Sixty-four percent met ^Ll three ^teria and were classified as 
typical. Based on this categorization, there were significant differences 
in performance on the TCS In-Basket exercise and the Total Score (f (). )C )) - 
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2.47, p = .015, and t fl'29) = 2.55, £ = .012, respectively), with atypical 
students scoring higher. differences between typical and atypical students 
were in the same direction far Group Discussion and Oral Presentation 
exercise scores, but not statistically significant. 

More important for present purposes than what group scored higher is 
that faqt that ICS scores did correlate with background and program variables. 
Only thg Group Discussion exercise secure did not correlate with any back- 




££j&xmd or program variable. \ ^ ^ 

Relati on sh ip s Between I _QS__and 
Other College P erf orm an c e Measure s 

Table 12 presents the partial correlations between performance on the 
three ICS exercises, and the ICS Total Score, and the other college per- 
formance measures, controlling for variability due to background \and program 
variables. Due to the variability in administration of the Integrated 
Competence Seminar to students in the two entering classes, it is not 
possible to place all indices in a clear temporal order. Generally, ^the 
ICS. scores may be regarded as concurrent with credit, semester, and compe- 
tence level unit information collected to coincide with the second longi- 
tudinal assessment of human potential measures. The first Six Performance 
Characteristics rating would also have occurred near the same time as ICS 
administrations. The data collected to coincide with the third and final 
longitudinal assessment would clearly have been collected after the. ICS 
was administered. 

In the combined Weekday College samples, the number of semesters 
attended by 'the second longitudinal assessment was correlated with major 
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Table 12. 



Partial Correlations of ICS Scores With Other 
College Performance Measures, Controlling 
For Background and Program Covariates 



Concurrent 

Measures 
Number of 
Semesters 
(8) 



Exerc ses 



Group Discussion 



.042 
(127) 



In-Basket 
(2) 



Total 



Oral Presentation Score 
( 3,7,8) (2,3,7) 



-. 252** 
(124) 



-.014 
(126) 



-.114 

(123) 



Credit Hours 
(1,7,8) 

Competence 
Level Units 
(4,5,7,8) 



-.187* 
(127) 

* Oil 

(92) 



.013 
(122) 

-.003 
(91) 



.04 7 
(125) 

-.082 
(93) 



-.070 
(122) 

-.044 
(91) 



Six Performance .134 
Characteristics (98) 

Rat ing 

(4,6) 



.180* 
(97) 



.082 
(96) 



.139 
(95) 



Later Measures 

Numfeer of .296*** 
Semesters (127) 
(-) 

Credit Hours .103 
(9) (127) 

•» 

Competence .044 
Level Units (126) 
(10) 

Six Performance .042 
Characteristics (98) 

Rating 

(4,1D 



.162* 
(126) 



.120 
(125) 

.117 
(124) 

-.005 - 
(97) 



.082 
(126) 

.030 
(125) 

-.053 
(124) 

.073 
(96) 



. 223** 
(124) 

.117 
(123) 

.057 
(122) 

.044 
(95) 



05 
.01 
.001 



Note. Numbers in \parentheses in cells are sample sizes. Numbers in parentheses 



in headings\ are covariates as follows 

1 — Age at* Entrance < 

2 — Religious Affiliation 

3 — Marital Status 

4 — High School Grade Point; Average 

5 — Mother's Occupation 



6 — Entrance Cohort 

7 ~- Full time /part time Status 

8 — Major Field 

9 — Credits Earned in First Interval 

10 — Competence Level. Units 

in First Interval 

11 — Prior Six Characteristics Rating 
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field. The number of credits accumulated by second assessment was related 
to fulltime status, major field, and age; competence level units accumulated 
were related to fulltiqp status, major field, high school grade average, 
and mother's occupation. The partial correlations presented in Table 12 
control for both these influeaces on semesters,^credits, and competence 
YeveJ. units, as well as background and program covariates of ICS performance 
-scores. The same is true for Six Performance Characteristics ratings, which 
were related to high school grade average and entrance cohort.,. 

The significant correlation beiWen number of semesters and the In- 
Basket .score indicated that students who attended more semesters tended tc 
receive lower .scores. Most students attended 'at least the four regular 
semesters falling between entrance and second assessment, so the variability 
in- number of semesters was limited to attendance during summer sessions. 
Thus, the data indicated that students who attended summer sessions tended 
to receive lower En-Basket exercise scores. There was a small but significant 
correlation between In-Basket scores and the first Six Jharacteristics 
rating. This correlation may be regarded as reciprocally supportive of the 
validity of the In-Basket exercise score as a measure of performance, and 
of the Six Characteristics ratings as a faculty rating of student performance. 

The significant correlation between number of credits accumulated and 
Group discussion scores indicated that students who accumulated more credits 
tended to receive lower scores on the Group Discussion exercise, with the 
influences of major, fulltime status, marital status and age controlled. 
In the case of the negative correlation between semesters and In-Basket 
scores, we may speculate that the association arises from the fact that 
students needing basic skills education and students re-taking a failed 
course populate the summer sessions (though we haven't the necessary data to 
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verify this guess). The negative correlation betw.i*i*n (initip hi unM** I mi MiMitci 

\ • *' 
and number of credits accumulated is more puzzling. 

\ *• • 

The only significant partial correlations between ICS scores and later 

measures. of tollege performance were with number of semesters between second 

and third assessment. The number of semesters in this interval^anged 

only between ithree and five, with nearly three-quarters of the students; 

attending the: normal four semesters. This limited range allows th^irtter- 

piietation that students who attended summer school between their third and 
I - \ 
fourth years recBived higher Group Discussion and In-Basket scores, a 

' r i 

finding for which there is no obvious explanation. \ 

V 1 . 



X 



The partial correlations in Table 12 indicate there is no single 



Y 



performance variable underlying the several, indices of college performance. 
'The correlations between Group Discussion exercise scores and credits and 
. semesters provided evidence that Group Discussion scores were not 
without.. some reliability, but do not necessarily . indicate validity for the 
intended purposes of the score as a performance measure. 

r 

Rela tionships Between ICS an d 
Human Pot ential Measures 

As stated earlier, the repeated administrations of the human potential 
measures at entrance, two years after entrance, and again three and one-half 
years after entrance, provided three sets of correlations with ICS scores 
in different temporal relationships. Tables 13, 14, and 15, present the' 
partial correlations between ICS scores and human potential, measures for, 1 i 
each temporal relationship, respectively. En each case, background and ' • 
program influences on ICS scores and human potential measures were controlled 



Table 13: 



Partial Correlations of Integrated Competence Seminar .Scores with Entrance 
Assessments on the Human Potei/tial Measures 



Exercises 

Croup 

Discussion ' In-Basket 
(2) 



Oral * Total 
Presentation Score 
(3,7,8) (2,3,7) 



Measures of Vocational, Educa- 
tional an^. Personal Issues 

"Best Class" Essay (8) .-.015 (118) 
"Decision"* Essay (1,2,5,8) .020 ( 95) 
"Career" Essay (1,4,8) .072 ( 92) 



Sentence Completion Test (8). 

Defining Issues Test 

P% Score (1) 
D Score (1) i 

Test of Cognitive Develop- 
ment (5,8) 



,015 (115) -.018 (116) 
,005 ( 95) .06*3 i- 93) 
.068 ( 89) .030 ( 90) 



139 (118) .258**(115) .102 (116) 



-.075 (119) :175* (116) .018 (117) 
-.182*(119) .147 (1-16)' .037 (117) 

■-.084 ( 95) .120 ( 93) -.025 ( 93) 



•.028 (114) • 
.065 ( 94) 
.071 ( 88) 

" Y 

.083 (114)> 



,065 (115) 
.020 (115) 

.044 ( 92) 



Picture Story Exercise 



Receptive (8).^ 




-.001 


(119) 


.017 


(116) . 


.046 (117) 


.015 (115) 


Autonomous 




-.100 


(122) 


.067 


(117) -; 


.087 (118) 


-.056 (116) 


Assertive (2,6) 




-.135 


(116) 


-.109 


(116) -. 


. 156*(114) 


-.179*0-15) 


Integrative (8,10) 




-.068 


(118) 


.017 


(116) -, 


.056 (117) 


-.037 (115) 


Self-Definition (8,10) 


.Oil 


(118) 


-.046 


(116) , 


.035 . (117) 


.008 (115) 


Achievement Motive 


(8,9) 


.034 


(118) 


.035 


(115) 


.141 (116) 


.099 (114) 


Affiliation Motive 


(8) 


.000 


(11^) 


.077 


(116) 


.179*(117) 


.106 (115) 


.Power Motive 




-,250**(122) 


.027 


(117) - 


,22&**(il8) 


-. 197*(116) 



Learning Style Inventory 



Concrete Experience (1) 
Reflective Observation 
Abstract Conceptuali- 


.030 (123) 
-.067 (126) 
-. 049JJ,2'3) 


-.I69*tl20) . 
-.057 (120) -. 
.152 (120) . 


.001 
.082 
.009 


(121) 
(122) 
(121) 


-.056 
-. Til 
-.071 


(119) 
(120) 
(IT 9) 


za Ion (1) 
Act' e Experimentation 
Abstract /Concrete 


-.014 
-.050 


(126) 
(123) 


-.009 (120) -. 
,197*(120) 


.114 
.006 


(122) 
(121) 


-.039 
.080~ 


(120) 
(119) 


Learning Orientation (1) 
Active /Reflective 


.035 


(126.) 


.088 (.121) - 


.010 


(122) 


.055 


(120.) 


Learning Orientation 
















Test of Thematic Analysis (8) 


-.'044 


(121) 


.216**U18)- 


.046 


(119) 


.098 


(117) 



**p 



.05 
.01 



Tab It 1 13. cant inueci 



c a 1. i i * lauding, ;-.ppr a i aa i 

Inference -.038 (.117) .2()6*(112) .120 (113) .119 (1U) 

Recognition -.180 ( 11.7) .073 (.1 1.2) . 2A8**(1 13) . 024 (11.1) 

Deduction (3,5) ~.223*( 76) .168 ( 75) .224*( 74) .045 ( 74) 

OS 



P 

*v .01. 



>U*. Numbers in parentheses in ca -is arc :. ample sizes. Numbers in parentheses 
in headings are covari;'tes .is fol Lows - 

1 — - Age 6 — Prior College Experience 

2 ~ Religious Affiliation 1 — Marital Status 

5 Father's Education -""8 — Entrance Cohort: 

--- Mother's ih/.eupnt i or. ~~ Residence 

5 — High School. Grade Point Average 10 — Fu 1 It ime /pa r t t i me 

11 Major 
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Table 14. 

Partial Correlations ot Integrated Competence Seminar Scores with Concurrent. 

Chain* *j on t lie Ilunian Potential Measu res - 



Lxerc ises 



Discussion 



Ln-Basket 1 



Oral Total 
'res en tat ion Score 









(2 ) 




.(3, 7 


.,8) 


(2,3,7) 


Measures <>! Vucat iwna 1 , Kduca- 


















t iona I. and Personal. I ssues 


















"Best Class" Essay 


-. 039 


(118) 


-.087 


(J 14) 


-.080 


(115) 


-. 100 


(113) 


"Decision 11 Essay (1) 


> L27 


(1 18) 


-..173' 


v ( 1 1 5 ) 


.055 


(116) 


. 0 1 1 


( I 14) 


"Career" Essay 


. 1.56 


( 1 03) 


. 02 2 


(100) 


. 02 0 


(101) 


. 1.04 


( 99) 


Sentence Completion Test (8) 


- . 00 L 


( 1 1 7 ) 


. 045 


( 114) 


. 062 


(115) 


. 02 5 


(111) 


Dei iniin; Issuos iVst 


















P, : : Score 


-.075 


(119) 


.120 


(116 ) 


. 165 


(-U7) 


.050 


(115) 


1) Score 


. 008 


(119) 


. 099 


(1 1 6) 


.108 


(J 17) 


. 059 


(115) 


Test of Cognitive Develop- 


. 04 2 


(116) 


. 159' 


HUi) 


. 146 


(1 14) 


. 1 1 4 


(112) 


ment (8) 


















Picture Story Exercise 


















R ec e p t i ve ( b ) 


. 008 


(118) 


nfr n 

. u o \ j 


MIS'! 
V -L -L 3 J 


. 05 5 


(116) 


. 04 8 


( 1 14 ) 


Autonomous 


.049 


(119) 


. 061 


( 116) 


-.041 


(117) 


t"\f i 

. Uot> 


( J ID) 


Assert ive 


-.002 


(119) 


. 113 


(lib) 


.016 


(117) 


. U J .) 


{ 1 1 D ) 


Integrative 


197*(119) 


. U J 1 


( IXo) 


-.009 


(117) 


- . .1 L O 


{ .1. J. D ) 


Self -De t mit ion (i0) 


-.024 


(118) 


. Uoo 


( 1 1 0 J 


-.023 


(117) 


n i 7 


\ ll D ) 


Achievement Motive. 


-.036 


(119) 


1 T 0 
. 11/ 


( LXO) 


. 15 , 3*(117) 


.081 


(1 15) 


Affiliation Motive (8) 


. 005 


(118) 


. 04 I 


(115) 


.07 7 


(116) 


.061 


(114) 


Power Motive 

j 


.060 


(119) 


. 109 


(116) 


-.014 


(117) 


.UOO 


c 1 1 S^i 
\11D ) 


Learning St vie Inventory 


















Concrete Experience 


-.176* 


'(123) 




(120) 


-.104 


(121) 


-.089 


(119) 


Reflective Observation 


- . 122 


(123) 


.oh 


(120) 


-.159*(121) 


-. 155'' 


H119) 


Abs t r a c t Con cep t ua 1 i z a 


.080 


(123) 


-110 


(120) 


.068 


(121)' 


.026 


(119) 


t ion 


















Ac t i v e Ex pe r imen t a t i o n 


.1.42 


(123) 


-.052 


(120) 


; 194*(121) 


. 130 


(119) 


Abstract / Concrete 


. 144 


(123) 


-.108 


(120) 


.096 


(121) 


. 060 


(119) 


Learning Orientation 


















Act ive /Re fleet ive 


. 147 


(123) 


.015 


(120) 


.196* 


: (121) 


.153 


(119) 


Learning Orientation 


















Test of Thematic Analysis (8) 


-.040 


(120) 


.088 


(117) 


-.031 


(118) 


.013 


(116) 



•p 



.05 
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1 ' vi b 1 e 1 4 . continued 



Critical Thinking Appraisal 



Inrerp.nce 
Recognition 
Deduction (5) 



-.091 (1.14) 
,086 (114) 
-.038 ( 92) 



. 1.A4 (111) .020 (112) 
. 141 (13 L) ,034 (112) 
.102 ( 90) .139 ( 90) 



.006 (110) 
.103 (110) 
.046 ( 89) 



See not e on b o 1 1 oni o f 1 a b .1. c 13 f or 1 e i\ c 1 1 d 
*p < .05 
**p < .01 
***p < .001 

Note. All human potential measures at second assessment were controlled for 
differences at en t ran c e . 



\ 
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Table 15. 



Partial. Correlations ui" integrated Competence Seminar Scores with 
Change on the Human Potential Measures 



h Lai or 



Measure of Vocational, Kduca- 
tional and Personal Issues 

"Best Class" Essay 
''Decision" Essay (2) 
"Career" Essay (8) 

Sentence Comp 1 et i on Test 

be lining Is sue s 1 e s t 

\ } Z Score 
D Score (1) 

Test of Cognitive Develop- 
ment 



Croup 
Discussion 



.J54*(U9) 
. 152* (3 1.6) 
-.065 (1.1.7) 



Exercises 

Oral 

J n-Hasket Presentation" 
(2) , .(3,7,8) 



.012 (115) -.028 (116) 
.062 (113) .007-4112) 
.005 (114) -.060 (1.15) 



Total 
S c o r e 
(2,3,7) 



,097 (114) 
046 (113)' 
,049 (113) 



,011 (118) -.005 (115) -.106 (116) -.057 (114) 



.077 (11.8) ,113 (115) .11 7 (116) .122 (114) 
-.042 (118) -.048 (115) .006 (116) -.030 (114; 

.070 (117) .205^(114) .037 (115) .146 (113) 



Picture Story Exercise 

Kecentive -.034 (118) .055 (115) -.102 (116) .017 (114) 

Autonomous -.026 (.118) .083 (115) .242**>116) .125 (114) 

Assertive -.003 (118) .042 (115) .114 (116) ,071 (114) 

Integrative .057 (118) -.090 (115) -.15^(116) -.080 (114) 

Self-Definition (8,10) -046 (117) .072 (115) -.007 (116) .053 (114) 

Achievement Motive .008 (119) -.029 (116) -.001 '(117) -.011 (115 

Affiliation Motive (8) .0.15 (118) -.024 (.115) -.045 (116) -.023 (iU) 

Power Motive .012 (119) .177*(116) .115 (117) .140 (115) 



Learning Style Inventory 



Concrete Experience 
'Ref le.-tive Observation 
Abstract Conceptuali- 


-.052 
-.100 
-.059 


(123) 
(123) 
(123) 


-.111 
-.129 
.075 


(120) 
(121) 
(120) 


-.013 
-.103 
-.081 


(121 s 
(121) 
(121) 


-.076 
-.078 
-.028 


(119) 
(119) 
(119)' 


zation 
Active Experimentation 
Abstract /Concrete 


. 012 
-.009 


(123) 
(123) 


.013 
. 106 


(121) 
(120) 


. 120 
-.04 3 


(122) 
(121.) 


.040 
.02 3 


(119) 
(119) 


Learning Orientation 
Active/Reflective 


. 065 


(123) 


-.049 


(120) 


.119 


.(120 


' .063 


(119) 


learning Orientation 


















of Thematic Analysis 


. .030 


(121) 


. 109 


(118) 


.200' 


*(119> 


.119 


(117) 



.05 
,01 
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Table 15. continued 



Critical Thinking Appraisal 

Inference .012 ( ■'. ) .086 (111) .089 (11 2) .068 (110) 

Recognition -.105 (114) -.086 (111) -.lib (11. -.141 (110) 

Deduction -.070 (114) .083 (111) .129 (112) , .056 (110) 

See note on bottom of Table 13 for K-.,ciid. 

*p < .05 
** p < .01 

***p < .001 

Note. All human potential measures at third assessment were controlled for 
differences at second assessment. 
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Tables 14 and L5 Inc. luck' com ro Ls tor preceding assessments on the same 
instruments, so that, in effect, the partial correlation with ICS scores 
represents the association of ICS scores with change on the human potential 
measures unexplained by the preceding assessment and any background or 
program covariates of change, 

Jable yj presents the partial correlations between ICS scores and 
entrance assessments on the human potential measures. Variability in 
performance on the. Integrated Competence Seminar or the human potential 
measures which could be attributed to background and program differences 
was controlled in all cases. Entrance cohort was a significant correlate 
of -.entrance assessments in the majority of cases shown, underscoring the 
importance of using multiple cohorts' in program evaluation and studies ol 
student change in college. Age was a significant correlate on three human 
potential measures despite the small variability in age in the Weekday 
College samples: on our measure of Perry's scheme of intellectual and 
ethical development, Rest's objective measure of moral development, and 
the measure of learning styles developed by David Roll). 

There were few significant partial correlations between entrance 
assessments on the human potential measures and the ICS exercise scores or 
ICS total score. The few small but significant partial correlations provide 
strongest support for the validity of the in~Bafcket exercise as a generic 
measure of performance in college. There is less, but some supportive 
evidence for the validity of the Oral Presentation exercise, and counter- 
evidence for the validity of the Group Discussion exercise. 
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In-Basket exercise scores were correlated with entrance measures of 
ego development, moral development, learning style, and critical thinking 
skills. Oral Presentation exercise scores were correlated with entrance 
measures of critical thinking skills, but were-also negatively correlated 
with measures of self-definition and power motivation. The latter relation- 
ships do not directly support or invalidate the Oral Presentation exercise 
sccrj. Croup Discussion exercise scores were negatively correlated with 
a measure of moral development, with power motivation, and with a measure oi 
critical thinking. These relationships indicate that students who did less 
well on the test of deductive thinking ability, Chose with lower scores on 
one measure of principled thinking, and students with less motivation to 
influence others, received higher scores on the Group Discussion exercise. 
This counter-evidence was not equivocated by any significant positive 
relationships. The data suggest that some entrance measures predict perform- 
ance on the In-Basket, and the Group Discussion. 

Tables 14 and 15 describe relationships between ICS exercise scores 
and total score and chajnge on the measures of human potential for two 
successive intervals ofVirae. Table 14 presents partial correlations between 
ICS scores and change on the human potential measures between entrance and 
second assessment, two years after entrance. This period approximates the 
general education sequence for most students in the sample and is most related 
to the time of student performance on the ICS. Table 15 presents the partial 
correlations between ICS scores and' change on the human potential measures 
between the second and third longitudinal assessments, a period approximating 
the pre-professional sequence for most students i-n the sample, and usually 
follows ICS performance. 



There were t'uw patterns of significant relationship evident, but as in 
the case of Table 13, what evidence there was provides strongest support for 
relationships between the [n-Basket exercise and less support for the Oral 
Presentation cxericse. The relationships between Group Discussion scores 
and change on the human potential measures were negative. Overall, the 
statistically significant relationships were small enough and few enough, 
given the number of correlations generated, that the weight of evidence does 
u-.» t. support the relationships of the Integrated Competence Seminar performance 
scores to change on the Human Potential Measures. 

Summary and Discussion 

Preliminary studies on the Integrated Competence Seminar indicated 
adequate reliability in the scoring procedure, and there was evidence of 
discrimination in individual performance. The relationships examined here 
between the ICS scores and other measures of student performance do not 
contribute much support for relationships to at least one if not two of 
the exercise scores. In our judgment, the 1 n-Basket exercise has been . - 
shown to be the most difficult and the most related to other measures, 
the Oral Presentation exercise next most difficult and next most related 
to other measures, and the Group Discussion exercise the least difficult 
and the least related to other generic measures of student performance in 
college. 

There are some possible/ explanations for these results. One has .to do 
with performance mode. Another has to do with where the instrument is in 
its development as ^measure of integration of anilities achieved by the 
midpoint of college. 
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Til e j n -- H a s k e t w a s most r e 1 a t. e d to per f o r m a n c e in the 1 e a r n i n g pro c e * s 
and to change on the measures of human potential. it is also a measure 
calling for problem-solving and critical thinking, college abilities more 
traditional in content. While the performance mode is nont radi t ional , 
it differs in mode from the Oral Presentation and Croup Discussion. in 
the ln-Baaket, a student performs with respect to a particular paper and 
pencil task. The student does take on a role, but that is also true of 
the other t: w< > ICS e xe r < *. 1 s e s . The 0 r a 1 P res en t a t ion and G r o up Di s c u s s i on 
involve a speech and participation in a group as performance modes. It 
could be that these more nont radi t iona 1 modes elicit aspects of Communications 
and Social Interaction not tapped by the human potential measures. 

On the other hand, the evidence for negative relationships between 
Group Discussion and the college performance and human potential ^measures 

could be evidence that the measure is ineffective and invalid. '* 

\ 

In fact, even before these analyses were complete, the Assessment 
Committee, the group of faculty responsible for the design and validity 
of the ICS, and external assessors, had raised questions about the accept- 
ability and adequacy of the instrument stimulus, particularly related to the 
performance elicited by Group Discussion^. They felt the group task did not 
elicit identification of complex issues or task development and solution 
as intended. An instrument revision process was undertaken after the data 
examined here were collected, so that a different instrument stimulus is 

in use now. While it would not be possible to directly follow up the impli- 
A 

Rations of- this analysis because of the change in instruments, the results 
have general implications for. future development and research, particularly 
on the use of the performance mcdes used in this study. 
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i"he findings of this research, in terms of the relative validity of the 
exercises, suggest continuing to use the In-Basket, and to strive for better 
means to elicit the important abilities targeted in the Group Discussion 
and to further examine the Oral Presentation exercise. The Oral Presenta- 
tion and Group Discussion exercises attempt to assess Communications and 
Social Interaction abilities thac have not been traditional goals of 
teaching Ln liberal, education, but are easily recognised as critically 
important abilities in the world of work. Further, the performance modes — 
speaking and interacting — are new assessment modes. 

A related study of Social Interaction competence (Friedman, Mentkowski, 
Deutseh, Shovar and Allen, 1982) found that instructed and uninstructed 
students did not differ in the Group Discussion exercise. ' But when a 
student performance from a more heterogeneous group than were available 
for the current study was subjected to a discriminant analysis, instructed 
and uninstructed groups did separate on Social Interaction ability. 

Thus, we may have the nucleus of an ability measure in Group Discussion 
as a performance mode. Only further research on the new ICS instrument 
stimulus will help answer this question. In sum, the study lands some 
support for the In-Basket, and suggests further research with Oral Presen- 
tation and Group Discussion exercises, While the current measure provided 
reliable results for diagnostic purposes^ the new Integrated Competence 
Seminar needs to be researched further to continue its development as an 
external criterion measure. 
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A 



As Often Frequently At 



INDEPENDENT JUDGEMENT 



AS 

Possible 
A 
I 



F 
I 



Least 
Once 

O 

1 



Not 

At 

All 

N 
I 



GROUP DISCUSSION 



INITIATION 



2. Initiative - Active efforts to influence events rather than passive 

acceptance 

INFLUENCE 

3. Persistence - tendency to stay with a problem or line of thought 

until the matter is settled 

4. Impact - Ability to create : Good Impression 

Attention and respect 
Show an air of confidence 

5. Social effectiveness - Ability to bring a group to accomplishment 

of its goal by directing and facilitating 
actions 

Respons e 

6. Listening - ability to understand what was significant in oral 

communication of others. 

7. Flexibility - Intellectual , emotional, social - ability to 

modify behavioral style to reach a goal. 

8. Adaptability - to new situations- Ability to assume the ro\e called 

for by a given situation. 



A F 0 N 



A F 0 N 

A F 0 N 

A F 0 N 

A F 0 N 



A F 0 N 



A F 0 N 



A F 0 N 



A F 0 N 



PROBLEM ANALYSIS 

9 . 'organization'*- effectiveness in seeking out pertinent data and 

organizing it to arrive at a credible conclusion 
(Research, budget, ABC plan) 

10. Raising of questions 
Decision Making 

11. Consideration and evelauation of alternatives deeisions, 
conclusions behavior 

12. Decisiveness - Readiness to make decisions or to render judgement A F 0 N 

Total 



A F 0 N 
A F 0 N 



A F O N 
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IN - BASKET 



Setting Priorities 1 0 



\ 



Complete Exercise 1 2 3 A 5 6 

INITIATIO N 

1. Clear convincing writing which does not distract the reader by 

technical errors. A F 0 N 

2. Initiative - Active efforts to influence events rather than 

passive acceptance. A F 0 N 

INFLUENCE 

3. Impact - ability to create : Good impression A F 0 N 

Command attention and respect A F 0 N 

Show an air of confidence A F 0 N 

RESPONSE 

4. Flexibility - Intellectual-, emotional, social - ability to modify 

behavioral style to reach a goal A F 0 N- 

5. Adaptability - to new situations 1 A F 0 N 

6. Reading - ability to understand what was significant in each 

item,. * „ A F 0 N 

PROBLRH ANALYSIS 

7. Concern for significant details A F 0 N 

8. Making predictions based on pertinent data ability and 
willingness to project future implications of present situations 

and or decisions. A F 0 N 

i 

DECISION MAKING 

! 

9. Consideration and evaluation of alternatives: Decisions, conclusions \ 
behavior A F .0 N \ 

10. Decisiveness - readiness to make decisions or to render judgement A F 6 N N 

Total 

% j 
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ORAL' PRESENTATION 



NAME 



INITIATION 



■» r\ i ~ , j . • "I -» -r- unbare honrlaKl p mrrmnni Pfl r1 on of message 

1. Oral expression — clear unaersLanudui.e tunuuniiLduiun \jl mcocja^c 


A 


F 


0 


N. 


INFLUENCE 


> 








2. Impact - ability to create : A good impression 


A 


F 


0 


N 


Command attention and respect 


A 


F 


0 


N 


Show an air of confidence 


A 


F 


0 


N 


RESPONSE 










3. Reading - ability to Understand what was significant in source 










materials 


A 


F 


0 


N 


U. Adaptability to new situations - ability to assume the role called 










i 

for by a given situation 


A 


F 


0 


N 



PROBLEM ANALYSI S ^ 

5. Organization - effectiveness in seeking out pertinent data 

and organizing it to arrive at a credible conclusion A F 0 N 

6. Concern - for significant detail - ability to cite and utilize 
whatever can provide clarification or evidence (e.g. examples, 

observations, facts distinctions relationship). A F 0 N 

DECISION MAKING 

7. Consideration and evaluation of alternatives : Decisions, 

Conclusions, Behavior. A F 0 N 

H, Decisiveness - readiness to make decisions or to render judgement A F 0 N 

9. Value clarification - ability to identify values in decision making A F 0 N 



Total 
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